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ABSTRACT 

The problem of major concern to the Physics Education 
Evaluation Project (P.E.E.P.) involved the improvement of university 
physics teaching and learning. The present paper describes 
instruments and procedures developed for systematic formative 
evaluation of physics lectures. The data was drawn from two sections 
of a first year university physics course. Guided by stake's 
evaluation framework, descriptive data about the lecture environment 
was collected and analyzed. At formative evaluation sessions 
immediately following the lecture, the instructor and an 
evaluator-observer processed the data and suggested improvements for 
subsequent lectures. The techniques are foreseen as potentially 
useful for improving teaching and learning in other lecture 
situations. (Author) 
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Abstract 

Formative Evaluation of Lectures: An 



Application of Stake ' s Evaluation 



Framework 



Walter W. Westphal, Philip G. B a shook, 
and Walter B. Boldt 



The problem of major concern to the Physics Edu- 
cation Evaluation Project (P.E.E.P.) involved the im- 
provement of university physics teaching and learning. 

The present paper describes instruments and procedures 
developed for systematic formative evaluation of physics 
lectures. The data was drawn from two sections of a 
first-year university physics course. Guided by Stake's 
evaluation framework, descriptive data about the lecture 
environment was collected and analyzed. At formative 
evaluation sessions immediately following the lectures, 
the instructor and an evaluator-observer processed the 
data and suggested improvements for subsequent lectures. 

The techniques are foreseen as potentially useful for 
improving teaching and learning in other lecture situations. 
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Formative Evaluation of Lectures: An 

Application of Stake ' s Evaluation Framework* * 

Walter W. Westphal, Philip G. Bashook , 
and Walter B. Boldt + 



Objectives of the inquiry 

The work reported here constitutes part of the Physics 
Education Evaluation Project (P.E.E.P. , 1970) undertaken during 
1969-1970 at the University of British Columbia. The problem of 
major concern to the project involved the improvement of physics 
teaching and lemming at the university level. The present paper 
describes instruments and procedures developed for systematic 
formative evaluation of university lectures. 

Formative evaluation of a lecture refers to the feeding back 
of data about the lecture to concerned individuals (i.e. the in- 
structor, the students) in order to improve ongoing teaching and 
learning. In the P.E.E.P. project, an attempt was made to utilize 
Robert Stake's (1967) evaluation framework as a guide in organi- 
zing and identifying descriptive data. Since Stake's framework 
is a recent addition to the evaluator's tool chest, the work re- 
ported here represents an initial application of the framework 
for formative evaluation of university lecturing. 

The first phase of this study was initiated in May, 1969, 
after lengthy discussions with the major instructor of the physics 
course on problems of educational evaluation, and was completed in 
December, 1969. it involved clarifying the descriptive categories 
of Stake's framework, establishing the course rationale, and 
developing instruments and procedures for formative lecture evalu- 
ation. The second phase, applying the instruments and procedures, 
was undertaken during the second term of the course and was com- 
pleted in May, 1970. Both phases were part of the initial stage 
of the P.E.E.P. study (Bashook, Boldt, Page, Westphal, 1971). 

Descriptions of the course environment constitute the 
Descriptive Matrix of Stake's Framework (Figure 1). Data from 
single or grouped lectures can be evaluated separately with the 
framework (miniature model ) or the complete lecture series for 
the course can be evaluated (global model) (Bashook, et al, 1971). 



+ Present addresses: W. Westphal - institut f\ir die Padagogik 

der Naturwissenschaften, Kiel University, Kiel, Olshausenstrasse 
40/60, West Germany; P. Bashook - University of Illinois College of 
Medicine, Center for Educational Development, 835 South Wolcott Aye. 
Chicago, Illinois 60612; W. Boldt - University of British Columbia, 
Faculty of Education, Vancouver 8, Canada. 

*The authors gratefully acknowledge the support and assistance 



- 2 - 



Techniques of obtaining data for each model differ. For the 
global model, methods must be devised for data collection con- 
cerning the well-articulated course rationale, for pre- and 
post-course measuring instruments in both the cognitive and 
affective domains, and for the collection of considerable 
background information about the students (See P.E.E.P., 1970, 
for example) . The methods described in this paper are for the 
"miniature model". 

Methods 



Instruments and procedures were devised to obtain data from 
four sources: (1) the instructor, (2) students, (3) evaluator- 
observer, and (4) lecture verbal and written events. The in- 
structor completed questionnaires describing his lecture intents 
before each lecture. He alr.o made available a copy of his notes 
and all written material. Student reactions to the lecture were 
gathered by a rating scale distributed randomly at each lecture 
and returned at the conclusion of the lecture. The evaluator- 
observer identified student interest levels using an observation 
instrument and recorded pertinent lecture events. Lectures were 
audio-taped and all written material presented during lectures 
retained. However, audio-tapes were only used as a reference 
source and did not contribute to the formative evaluation pro- 
cedures . 

After reviewing the student ratings and the observational 
instrument records, the instructor and evaluator-observer dis- 
cussed the lecture events. These formative evaluation sessions 
usually occurred immediately following the lecture. By applying 
the “miniature model" (Figure 2), the instructor and evalutor 
tried to identify contingencies and congruencies in the lecture 
events which were useful for deciding upon improvements in the 
lecture environment. During the sessions, suggestions were made 
concerning subsequent lectures, and strengths and weaknesses in 
the educational setting were discussed. 

An important consideration throughout the evaluation process 
was maintaining a close cooperation between the instructor and the 
evaluator in order to insure thorough understanding of the useful- 
ness and limitations of the instruments and procedures. Moreover, 
students were encouraged to participate in course decision-making 
and to ask questions about the evaluation project. 
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Data sources 



Two sections of a first-year university physics course 
taught by the same instructor formed the data source for the 
study. Data from additional courses in chemistry, physics, and 
education were used for refining some of the instruments. The 
physics course presented general physics content, but the sections 
used different textbooks and applied different teaching strategies. 
There were 419 students enrolled in the lecture section used as a 
data source in the first stage of the study. The other section 
had 213 students enrolled. Each section met for one hour three 
times a week and once each week alternately for a two-hour tutorial 
session (discussion) or a three-hour laboratory session. The 
lectures were held in a theater-like hall containing standard 
lecturing facilities. 

Results 



Individual lectures were evaluated using the "miniature 
model" of Stake's Framework (Figure 2). Lecture intents, in 
terms of antecedents, transactions, and outcomes were obtained 
from the instructor by means of a Lecture Pre-Analysis Question- 
naire (Figure 3 ) . The instructor completed the questionnaire 
prior to giving the lecture, usually when he was preparing 
material for presentation. Among other things, he was asked to 
identify the specific purpose or purposes of each lecture event 
or transaction. Depending on the nature of the lecture and the 
number of purposes for each transaction, the instructor completed 
between 8 and 10 questionnaires per lecture. Further information 
about the instructor's intents was obtained from his lecture 
notes. The "Lecture Pre-Analysis Questionnaire" also served as 
a source of data for each lecture rationale. The specific pur- 
poses stated in the questionnaire were assumed to be his lecture 
goals. Additional lecture goals were teased from the instructor's 
course rationale. The data from these sources feed into the 
"miniature model" as shown in Figure 2. 

Observational data describing observed lecture antecedents, 
transactions, and outcomes for a lecture were obtained in four 
ways. A Student Response Questionnaire (Figure 4) distributed 
randomly to 10-15 students each lecture yielded information about 
students' observations of lecture antecedents, transactions, and 
outcomes. Ihe instructor's verbal and non-verbal comments served 
as the second source of observational data. An observer using the 
Lecture Observation Instrument (Figure 5) constituted another data 
source for lecture transactions and outcomes. Finally, information 
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about lecture content was obtained from lecture audio-tapes and 
copies of written materials including homework assignments and 
tests. Figure 2 shows schematically where observational data 
fit into the miniature model. 



Since the evaluator could not be painted invisible, he served 
as an additional change-agent for the course. The close coopera- 
tion between evaluator and clients (instructor and students) seemed 
to facilitate the instructor's attempts to improve the educational 
climate in the lectures. An important effect of the evaluator in 
this role was his influence in. getting the instructor to focus 
upon specific teaching or learning activities which seemed bene- 
ficial to the students and should be retained or events which 
seemed counter to the instructor 1 s goals and needed to be recon- 
sidered. In addition, based upon educational science, the evalua- 
tor suggested alternate teaching actions which would be more likely 
to lead to the instructor's specified goals. 

Besides teaching, the instructor served in the role of 
evaluator by helping develop formative evaluation instruments and 
mechanisms. As a result, the instructor was more attuned to the 
goals of evaluation and more willing to adopt some of the 
evaluator's suggestions. A question unanswered by the study is 
whether such a double role might be necessary for a successful 
application of the model. 



The students initially seemed hesitant to accept the pre- 
sence of an outsider in the lecture. However, once his role in 
the lecture was explained and the purpose of the P.E.E.P. project 
described, most students took an active part in any positive 
attempts at altering the lecturing situation. The students viewed 
the "Student Response Questionnaire" as an important channel for 
communication with the instructor. Frequently, students would 
make comments to the instructor via the questionnaire about their 
concerns. Just as frequently, students would request an opportunity 
to complete a questionnaire. 

Educational importance 




The arbitrariness of designing lectures can be greatly 
lessened by introducing data-gathering instruments, procedures, 
and a framework which helps systematize the descriptive data re- 
quired for decision-making. The instruments and procedures de- 
scribed in this paper demonstrate a successful application of 
Stake's Framework to the problem of systematic formative evaluation 
of Physics lectures. These techniques are foreseen as potentially 
useful for improving teaching and learning in any setting where 
lecturing is the dominant teaching format. In conclusion, it 
should be noted that the study described here is only one facet of 
an initial attempt to systematically describe and judge a college 
course . 
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Figure 1 . Rationale and Descriptive Matrix of Stake ' s 
Evaluation Framework 
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FIGURE 2. Instruments for Evaluating a Lecture Using Stake's 
Categories for the Miniature Model 
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